Road profile along a predicted path

ABSTRACT

Systems and methods are provided for determining a road profile along a predicted path. In one implementation, a system includes at least one image capture device configured to acquire a plurality of images of an area in a vicinity of a user vehicle; a data interface; and at least one processing device configured to receive the plurality of images captured by the image capture device through the data interface; and compute a profile of a road along one or more predicted paths of the user vehicle. At least one of the one or more predicted paths is predicted based on image data.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. Nonprovisional applicationSer. No. 15/151,135, filed May 10, 2016, which claims the benefit ofpriority of U.S. Provisional Patent Application No. 62/159,298, filed onMay 10, 2015, and U.S. Provisional Patent Application No. 62/189,338,filed on Jul. 7, 2015. The disclosures of the foregoing applications areincorporated herein by reference in their entireties.

BACKGROUND

Active or adaptive suspension systems for vehicles can be useful forimproving the ride quality of the vehicles and the comfort ofpassengers. Typically, active and adaptive suspension systems activelycontrol the operation of one or more elements of the vehicle'ssuspension system and thus change its behavior. For example, some activesuspension system change the vertical movement of the wheels relative tothe chassis of the vehicle body, or change the shock absorber stiffness.Reactive suspension simply reacts to the road surface or to obstacles onthe road without changing the behavior of the suspension system or itscomponents.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for determining a road profile along a predicted path.

Consistent with a disclosed embodiment, a system includes at least oneimage capture device configured to acquire a plurality of images of anarea in a vicinity of a user vehicle; a data interface; and at least oneprocessing device configured to receive the plurality of images capturedby the image capture device through the data interface; and compute aprofile of a road along one or more predicted paths of the user vehicle.At least one of the one or more predicted paths is predicted based onimage data.

Consistent with another disclosed embodiment, a method of estimating aroad profile includes acquiring a plurality of images of an area in avicinity of a user vehicle; and obtaining one or more predicted pathsfor the user vehicle. At least one of the one or more predicted paths ispredicted based on the plurality of images. The method further includescomputing a profile of a road along one or more predicted paths of theuser vehicle.

Consistent with other disclosed embodiments, non-transitory computerreadable storage media may store program instructions, which areexecuted by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a diagrammatic representation of an exemplary systemconsistent with the disclosed embodiments.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments.

FIG. 2B is a diagrammatic top view representation of the vehicle andsystem shown in FIG. 2A consistent with the disclosed embodiments.

FIG. 2C is a diagrammatic top view representation of another embodimentof a vehicle including a system consistent with the disclosedembodiments.

FIG. 2D is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2E is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 3A is a diagrammatic representation of an interior of a vehicleincluding a rearview mirror and a user interface for a vehicle imagingsystem consistent with the disclosed embodiments.

FIG. 3B is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 3C is an illustration of the camera mount shown in FIG. 3B from adifferent perspective consistent with the disclosed embodiments.

FIG. 3D is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 4 is a flowchart illustration of a method of providing a roadprofile along a predicted path of a vehicle, according to examples ofthe presently disclosed subject matter.

FIGS. 5A-5E are a sequence of images of a road ahead of a user vehicleand two predicted paths overlaid thereon, according to examples of thepresently disclosed subject matter, and

FIG. 6 is an example of an output road profile along a predicted path,in accordance with examples of the presently disclosed subject matter.

DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples.

Disclosed embodiments provide systems and methods that can be used aspart of or in combination with active suspension, adaptive suspension,autonomous navigation/driving and/or driver assist technology features.Driver assist technology refers to any suitable technology to assistdrivers in the navigation and/or control of their vehicles, such as FCW,LDW and TSR, as opposed to fully autonomous driving. Active and adaptivesuspension technology relates to any suitable technology to activelycontrol the operation of one or more elements of the vehicle'ssuspension system and thus change its behavior. In various embodiments,the system may include one, two, or more cameras mountable in a vehicleand an associated processor that monitor the environment of the vehicle.In further embodiments, additional types of sensors can be mounted inthe vehicle and can be used in the active/adaptive suspension and/orautonomous navigation and/or driver assist system. In some examples ofthe presently disclosed subject matter, the system may providetechniques for processing images of an environment ahead of a vehicle tocompute a profile of a road along one or more predicted paths of theuser vehicle.

In accordance with an aspect of the present disclosure, there isprovided a system comprising at least one image capture device, a datainterface and at least one processing device, configured to receiveimages captured by the image capture device through the data interface.The at least one image capture device configured to acquire a pluralityof images of an area in a vicinity of a user vehicle. The at least oneprocessing device is configured to compute a profile of a road along oneor more predicted paths of the user vehicle, where at least one of theone or more predicted paths is predicted based on image data.Optionally, the profile of the road along each one of the one or morepredicted paths of the user vehicle can be computed along the estimatedwheel tracks of the user vehicle along the respective predicted path.Still further by way of example, the profile of the road along each oneof the one or more predicted paths of the user vehicle can be limited toapproximately the width of the track of each wheel or wheel-pair (orgroup of wheels for trucks and other such multi-wheeled vehicles) of thevehicle along the respective predicted path.

In accordance with a further aspect of the presently disclosedembodiments, there is provided a method of computing a road profile. Inone embodiment, the method can include: capturing a plurality of imagesof an area in a vicinity of a user vehicle; obtaining one or morepredicted paths of the user vehicle; and computing a profile of a roadalong one or more predicted paths of the user vehicle, where at leastone of the one or more predicted paths is predicted based on image data.

FIG. 1, to which reference is now made, is a block diagramrepresentation of a system according to examples of the disclosedembodiments. System 100 can include various components depending on therequirements of a particular implementation. In some examples, system100 can include a processing unit 110, an image acquisition unit 120 andone or more memory units 140, 150. Processing unit 110 can include oneor more processing devices. In some embodiments, processing unit 110 caninclude an application processor 180, an image processor 190, or anyother suitable processing device. Similarly, image acquisition unit 120can include any number of image acquisition devices and componentsdepending on the requirements of a particular application. In someembodiments, image acquisition unit 120 can include one or more imagecapture devices (e.g., cameras), such as image capture device 122, imagecapture device 124, and image capture device 126. Optionally, system 100can also include a data interface 128 communicatively connectingprocessing unit 110 to image acquisition device 120. For example, datainterface 128 can include any wired and/or wireless link or links fortransmitting image data acquired by image acquisition device 120 toprocessing unit 110.

Both application processor 180 and image processor 190 can includevarious types of processing devices. For example, either or both ofapplication processor 180 and image processor 190 can include one ormore microprocessors, preprocessors (such as image preprocessors),graphics processors, central processing units (CPUs), support circuits,digital signal processors, integrated circuits, memory, or any othertypes of devices suitable for running applications and for imageprocessing and analysis. In some embodiments, application processor 180and/or image processor 190 can include any type of single or multi-coreprocessor, mobile device microcontroller, central processing unit, etc.Various processing devices can be used, including, for example,processors available from manufacturers such as Intel®, AMD®, etc. andcan include various architectures (e.g., x86 processor, ARM®, etc.).

Optionally, application processor 180 and/or image processor 190 caninclude any of the EyeQ series of processor chips available fromMobileye®. These processor designs each include multiple processingunits with local memory and instruction sets. Such processors mayinclude video inputs for receiving image data from multiple imagesensors and may also include video out capabilities. In one example, theEyeQ2® uses 90 nm-micron technology operating at 332 Mhz. The EyeQ2®architecture has two floating point, hyper-thread 32-bit RISC CPUs(MIPS32® 34K® cores), five Vision Computing Engines (VCE), three VectorMicrocode Processors (VMP®), Denali 64-bit Mobile DDR Controller,128-bit internal Sonics Interconnect, dual 16-bit Video input and 18-bitVideo output controllers, 16 channels DMA and several peripherals. TheMIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the secondMIPS34K CPU and the multi-channel DMA as well as the other peripherals.The five VCEs, three VMP® and the MIPS34K CPU can perform intensivevision computations required by multi-function bundle applications. Inanother example, the EyeQ3®, which is a third generation processor andis six times more powerful that the EyeQ2®, can be used in the disclosedexamples. In yet another example, the EyeQ4®, the fourth generationprocessor, can be used in the disclosed examples.

While FIG. 1 depicts two separate processing devices included inprocessing unit 110, more or fewer processing devices can be used. Forexample, in some examples, a single processing device may be used toaccomplish the tasks of application processor 180 and image processor190. In other embodiments, these tasks can be performed by more than twoprocessing devices.

Processing unit 110 can include various types of devices. For example,processing unit 110 may include various devices, such as a controller,an image preprocessor, a central processing unit (CPU), supportcircuits, digital signal processors, integrated circuits, memory, or anyother types of devices for image processing and analysis. The imagepreprocessor can include a video processor for capturing, digitizing andprocessing the imagery from the image sensors. The CPU can include anynumber of microcontrollers or microprocessors. The support circuits canbe any number of circuits generally well known in the art, includingcache, power supply, clock and input-output circuits. The memory canstore software that, when executed by the processor, controls theoperation of the system. The memory can include databases and imageprocessing software. The memory can include any number of random accessmemories, read only memories, flash memories, disk drives, opticalstorage, removable storage and other types of storage. In one instance,the memory can be separate from the processing unit 110. In anotherinstance, the memory can be integrated into the processing unit 110.

Each memory 140, 150 can include software instructions that whenexecuted by a processor (e.g., application processor 180 and/or imageprocessor 190), can control operation of various aspects of system 100.These memory units can include various databases and image processingsoftware. The memory units can include random access memory, read onlymemory, flash memory, disk drives, optical storage, tape storage,removable storage and/or any other types of storage. In some examples,memory units 140, 150 can be separate from the application processor 180and/or image processor 190. In other embodiments, these memory units canbe integrated into application processor 180 and/or image processor 190.

Optionally, the system can include a position sensor 130. The positionsensor 130 can include any type of device suitable for determining alocation associated with at least one component of system 100. In someembodiments, position sensor 130 can include a GPS receiver. Suchreceivers can determine a user position and velocity by processingsignals broadcasted by global positioning system satellites. Positioninformation from position sensor 130 can be made available toapplication processor 180 and/or image processor 190.

Optionally, the system 100 can be operatively connectible to varioussystems, devices and units onboard a vehicle in which the system 100 canbe mounted, and through any suitable interfaces (e.g., a communicationbus) the system 100 can communicate with the vehicle's systems. Examplesof vehicle systems with which the system 100 can cooperate include: athrottling system, a braking system, a suspension system and a steeringsystem.

Optionally, the system 100 can include a user interface 170. Userinterface 170 can include any device suitable for providing informationto or for receiving inputs from one or more users of system 100. In someembodiments, user interface 170 can include user input devices,including, for example, a touchscreen, microphone, keyboard, pointerdevices, track wheels, cameras, knobs, buttons, etc. With such inputdevices, a user may be able to provide information inputs or commands tosystem 100 by typing instructions or information, providing voicecommands, selecting menu options on a screen using buttons, pointers, oreye-tracking capabilities, or through any other suitable techniques forcommunicating information to system 100. Information can be provided bythe system 100, through the user interface 170, to the user in a similarmanner.

Optionally, the system 100 can include a map database 160. The mapdatabase 160 can include any type of database for storing digital mapdata. In some examples, map database 160 can include data relating to aposition, in a reference coordinate system, of various items, includingroads, lanes and layout of lanes, objects on the road, water features,geographic features, points of interest, etc. Map database 160 can storenot only the locations of such items, but also descriptors relating tothose items, including, for example, names associated with any of thestored features. In some embodiments, map database 160 can be physicallylocated with other components of system 100. Alternatively oradditionally, map database 160 or a portion thereof can be locatedremotely with respect to other components of system 100 (e.g.,processing unit 110). In such embodiments, information from map database160 can be downloaded over a wired or wireless data connection to anetwork (e.g., over a cellular network and/or the Internet, etc.).

Image capture devices 122, 124, and 126 can each include any type ofdevice suitable for capturing at least one image from an environment.Moreover, any number of image capture devices can be used to acquireimages for input to the image processor. Some examples of the presentlydisclosed subject matter can include or can be implemented with only asingle-image capture device, while other examples can include or can beimplemented with two, three, or even four or more image capture devices.Image capture devices 122, 124, and 126 will be further described withreference to FIGS. 2B-2E, below.

It would be appreciated that the system 100 can include or can beoperatively associated with other types of sensors, including forexample: an acoustic sensors, a RF sensor (e.g., radar transceiver), aLIDAR sensor. Such sensors can be used independently of or incooperation with the image acquisition device 120. For example, the datafrom the radar system (not shown) can be used for validating theprocessed information that is received from processing images acquiredby the image acquisition device 120, e.g., to filter certain falsepositives resulting from processing images acquired by the imageacquisition device 120 or for augmenting, completing or otherwiseimproving images acquired by the image acquisition device 120.

System 100, or various components thereof, can be incorporated intovarious different platforms. In some embodiments, system 100 may beincluded on a vehicle 200, as shown in FIG. 2A. For example, vehicle 200can be equipped with a processing unit 110 and any of the othercomponents of system 100, as described above relative to FIG. 1. Whilein some embodiments vehicle 200 can be equipped with only a single-imagecapture device (e.g., camera), in other embodiments, such as thosediscussed in connection with FIGS. 2B-2E, multiple image capture devicescan be used. For example, either of image capture devices 122 and 124 ofvehicle 200, as shown in FIG. 2A, can be part of an ADAS (AdvancedDriver Assistance Systems) imaging set.

The image capture devices included on vehicle 200 as part of the imageacquisition unit 120 can be positioned at any suitable location. In someembodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122can be located in the vicinity of the rearview mirror. This position mayprovide a line of sight similar to that of the driver of vehicle 200,which can aid in determining what is and is not visible to the driver.

Other locations for the image capture devices of image acquisition unit120 can also be used. For example, image capture device 124 can belocated on or in a bumper of vehicle 200. Such a location can beespecially suitable for image capture devices having a wide field ofview. The line of sight of bumper-located image capture devices can bedifferent from that of the driver. The image capture devices (e.g.,image capture devices 122, 124, and 126) can also be located in otherlocations. For example, the image capture devices may be located on orin one or both of the side mirrors of vehicle 200, on the roof ofvehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, onthe sides of vehicle 200, mounted on, positioned behind, or positionedin front of any of the windows of vehicle 200, and mounted in or nearlight figures on the front and/or back of vehicle 200, etc. The imagecapture unit 120, or an image capture device that is one of a pluralityof image capture devices that are used in an image capture unit 120, canhave a FOV that is different than the FOV of a driver of a vehicle, andnot always see the same objects. In one example, the FOV of the imageacquisition unit 120 can extend beyond the FOV of a typical driver andcan thus image objects which are outside the FOV of the driver. In yetanother example, the FOV of the image acquisition unit 120 is someportion of the FOV of the driver, optionally, the FOV of the imageacquisition unit 120 corresponding to a sector which covers an area of aroad ahead of a vehicle and possibly also surroundings of the road.

In addition to image capture devices, vehicle 200 can be include variousother components of system 100. For example, processing unit 110 may beincluded on vehicle 200 either integrated with or separate from anengine control unit (ECU) of the vehicle. Vehicle 200 may also beequipped with a position sensor 130, such as a GPS receiver and may alsoinclude a map database 160 and memory units 140 and 150.

FIG. 2A is a diagrammatic side view representation of a vehicle imagingsystem according to examples of the disclosed embodiments. FIG. 2B is adiagrammatic top view illustration of the example shown in FIG. 2A. Asillustrated in FIG. 2B, the disclosed examples can include a vehicle 200including in its body a system 100 with a first image capture device 122positioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200, a second image capture device 124 positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200, and aprocessing unit 110.

As illustrated in FIG. 2C, image capture devices 122 and 124 may both bepositioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200. Additionally, while two image capture devices 122 and124 are shown in FIGS. 2B and 2C, it should be understood that otherembodiments may include more than two image capture devices. Forexample, in the embodiments shown in FIGS. 2D and 2E, first, second, andthird image capture devices 122, 124, and 126, are included in thesystem 100 of vehicle 200.

As illustrated in FIG. 2D, image capture device 122 may be positioned inthe vicinity of the rearview mirror and/or near the driver of vehicle200, and image capture devices 124 and 126 may be positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200. And asshown in FIG. 2E, image capture devices 122, 124, and 126 may bepositioned in the vicinity of the rearview mirror and/or near the driverseat of vehicle 200. The disclosed examples are not limited to anyparticular number and configuration of the image capture devices, andthe image capture devices may be positioned in any appropriate locationwithin and/or on vehicle 200.

It is also to be understood that disclosed embodiments are not limitedto a particular type of vehicle 200 and may be applicable to all typesof vehicles including automobiles, trucks, trailers, motorcycles,bicycles, self-balancing transport devices and other types of vehicles.

The first image capture device 122 can include any suitable type ofimage capture device. Image capture device 122 can include an opticalaxis. In one instance, the image capture device 122 can include anAptina M9V024 WVGA sensor with a global shutter. In another example, arolling shutter sensor can be used. Image acquisition unit 120, and anyimage capture device which is implemented as part of the imageacquisition unit 120, can have any desired image resolution. Forexample, image capture device 122 can provide a resolution of 1280×960pixels and can include a rolling shutter.

Image acquisition unit 120, and any image capture device which isimplemented as part of the image acquisition unit 120, can includevarious optical elements. In some embodiments one or more lenses can beincluded, for example, to provide a desired focal length and field ofview for the image acquisition unit 120, and for any image capturedevice which is implemented as part of the image acquisition unit 120.In some examples, an image capture device which is implemented as partof the image acquisition unit 120 can include or be associated with anyoptical elements, such as a 6 mm lens or a 12 mm lens, for example. Insome examples, image capture device 122 can be configured to captureimages having a desired field-of-view (FOV) 202, as illustrated in FIG.2D.

The first image capture device 122 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

FIG. 2F is a diagrammatic representation of vehicle control systems,according to examples of the presently disclosed subject matter. Asindicated in FIG. 2F, vehicle 200 can include throttling system 220,braking system 230, and steering system 240. System 100 can provideinputs (e.g., control signals) to one or more of throttling system 220,braking system 230, suspension system 245, and steering system 240 overone or more data links (e.g., any wired and/or wireless link or linksfor transmitting data). For example, based on analysis of imagesacquired by image capture devices 122, 124, and/or 126, system 100 canprovide control signals to one or more of throttling system 220, brakingsystem 230, and steering system 240 to navigate vehicle 200 (e.g., bycausing an acceleration, a turn, a lane shift, etc.). Further, system100 can receive inputs from one or more of throttling system 220,braking system 230, and steering system 240 indicating operatingconditions of vehicle 200 (e.g., speed, whether vehicle 200 is brakingand/or turning, etc.).

As shown in FIG. 3A, vehicle 200 may also include a user interface 170for interacting with a driver or a passenger of vehicle 200. Forexample, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver orpassenger of vehicle 200 may also use handles (e.g., located on or nearthe steering column of vehicle 200 including, for example, turn signalhandles), buttons (e.g., located on the steering wheel of vehicle 200),and the like, to interact with system 100. In some embodiments,microphone 350 may be positioned adjacent to a rearview mirror 310.Similarly, in some embodiments, image capture device 122 may be locatednear rearview mirror 310. In some embodiments, user interface 170 mayalso include one or more speakers 360 (e.g., speakers of a vehicle audiosystem). For example, system 100 may provide various notifications(e.g., alerts) via speakers 360.

FIGS. 3B-3D are illustrations of an exemplary camera mount 370configured to be positioned behind a rearview mirror (e.g., rearviewmirror 310) and against a vehicle windshield, consistent with disclosedembodiments. As shown in FIG. 3B, camera mount 370 may include imagecapture devices 122, 124, and 126. Image capture devices 124 and 126 maybe positioned behind a glare shield 380, which may be flush against thevehicle windshield and include a composition of film and/oranti-reflective materials. For example, glare shield 380 may bepositioned such that it aligns against a vehicle windshield having amatching slope. In some embodiments, each of image capture devices 122,124, and 126 may be positioned behind glare shield 380, as depicted, forexample, in FIG. 3D. The disclosed embodiments are not limited to anyparticular configuration of image capture devices 122, 124, and 126,camera mount 370, and glare shield 380. FIG. 3C is an illustration ofcamera mount 370 shown in FIG. 3B from a front perspective.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of system 100. Further, any componentmay be located in any appropriate part of system 100 and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and, in response to thisanalysis, navigate and/or otherwise control and/or operate vehicle 200.Navigation, control, and/or operation of vehicle 200 may includeenabling and/or disabling (directly or via intermediary controllers,such as the controllers mentioned above) various features, components,devices, modes, systems, and/or subsystems associated with vehicle 200.Navigation, control, and/or operation may alternately or additionallyinclude interaction with a user, driver, passenger, passerby, and/orother vehicle or user, which may be located inside or outside vehicle200, for example by providing visual, audio, haptic, and/or othersensory alerts and/or indications.

As discussed below in further detail and consistent with variousdisclosed embodiments, system 100 may provide a variety of featuresrelated to autonomous driving, semi-autonomous driving and/or driverassist technology. For example, system 100 may analyze image data,position data (e.g., GPS location information), map data, speed data,and/or data from sensors included in vehicle 200. System 100 may collectthe data for analysis from, for example, image acquisition unit 120,position sensor 130, and other sensors. Further, system 100 may analyzethe collected data to determine whether or not vehicle 200 should take acertain action, and then automatically take the determined actionwithout human intervention. It would be appreciated that in some cases,the actions taken automatically by the vehicle are under humansupervision, and the ability of the human to intervene adjust abort oroverride the machine action is enabled under certain circumstances or atall times. For example, when vehicle 200 navigates without humanintervention, system 100 may automatically control the braking,acceleration, and/or steering of vehicle 200 (e.g., by sending controlsignals to one or more of throttling system 220, braking system 230, andsteering system 240). Further, system 100 may analyze the collected dataand issue warnings, indications, recommendations, alerts, orinstructions to a driver, passenger, user, or other person inside oroutside of the vehicle (or to other vehicles) based on the analysis ofthe collected data. Additional details regarding the various embodimentsthat are provided by system 100 are provided below.

Multi-Imaging System

As discussed above, system 100 may use a single or a multi-camerasystem. The multi-camera system may use one or more cameras facing inthe forward direction of a vehicle. In other embodiments, themulti-camera system may include one or more cameras facing to the sideof a vehicle or to the rear of the vehicle. In one embodiment, forexample, system 100 may use a two-camera imaging system, where a firstcamera and a second camera (e.g., image capture devices 122 and 124) maybe positioned at the front and/or the sides of a vehicle (e.g., vehicle200). The first camera may have a field of view that is greater than,less than, or partially overlapping with, the field of view of thesecond camera. In addition, the first camera may be connected to a firstimage processor to perform monocular image analysis of images providedby the first camera, and the second camera may be connected to a secondimage processor to perform monocular image analysis of images providedby the second camera. The outputs (e.g., processed information) of thefirst and second image processors may be combined. In some embodiments,the second image processor may receive images from both the first cameraand second camera to perform stereo analysis. In another embodiment,system 100 may use a three-camera imaging system where each of thecameras has a different field of view. Such a system may, therefore,make decisions based on information derived from objects located atvarying distances both forward and to the sides of the vehicle.References to monocular image analysis may refer to instances whereimage analysis is performed based on images captured from a single pointof view (e.g., from a single camera). Stereo image analysis may refer toinstances where image analysis is performed based on two or more imagescaptured with one or more variations of an image capture parameter. Forexample, captured images suitable for performing stereo image analysismay include images captured: from two or more different positions, fromdifferent fields of view, using different focal lengths, along withparallax information, etc.

For example, in one embodiment, system 100 may implement a three cameraconfiguration using image capture devices 122-126. In such aconfiguration, image capture device 122 may provide a narrow field ofview (e.g., 34 degrees, or other values selected from a range of about20 to 45 degrees, etc.), image capture device 124 may provide a widefield of view (e.g., 150 degrees or other values selected from a rangeof about 100 to about 180 degrees), and image capture device 126 mayprovide an intermediate field of view (e.g., 46 degrees or other valuesselected from a range of about 35 to about 60 degrees). In someembodiments, image capture device 126 may act as a main or primarycamera. Image capture devices 122-126 may be positioned behind rearviewmirror 310 and positioned substantially side-by-side (e.g., 6 cm apart).Further, in some embodiments, as discussed above, one or more of imagecapture devices 122-126 may be mounted behind glare shield 380 that isflush with the windshield of vehicle 200. Such shielding may act tominimize the impact of any reflections from inside the car on imagecapture devices 122-126.

In another embodiment, as discussed above in connection with FIGS. 3Band 3C, the wide field of view camera (e.g., image capture device 124 inthe above example) may be mounted lower than the narrow and main fieldof view cameras (e.g., image devices 122 and 126 in the above example).This configuration may provide a free line of sight from the wide fieldof view camera. To reduce reflections, the cameras may be mounted closeto the windshield of vehicle 200, and may include polarizers on thecameras to damp reflected light.

A three camera system may provide certain performance characteristics.For example, some embodiments may include an ability to validate thedetection of objects by one camera based on detection results fromanother camera. In the three camera configuration discussed above,processing unit 110 may include, for example, three processing devices(e.g., three EyeQ series of processor chips, as discussed above), witheach processing device dedicated to processing images captured by one ormore of image capture devices 122-126.

In a three camera system, a first processing device may receive imagesfrom both the main camera and the narrow field of view camera, andperform processing of the narrow FOV camera or even a cropped FOV of thecamera, such as a region of interest (ROI) within the field of view ofthe camera. In addition, one the same or different images, severalprocessing operations can be performed, including on different parts orportions of the images, and optionally some processing operation can beapplied to the full image whereas other processing operation can useonly a portion of the (same) image. For example, the first processingdevice can be configured to and used for computing a profile of a roadalong one or more predicted paths of the user vehicle. In yet anotherexample, the first processing device can be configured to and used fordetermining a predicted path of the vehicle, using image data obtainedfrom an image capture device onboard the vehicle. Still further by wayof example, the first processing device can be configured to use atrained neural network to estimate a predicted path ahead of a currentlocation of the vehicle, in accordance with examples of the presentlydisclosed subject matter.

The first processing device can be further adapted to preform imageprocessing tasks, for example, which can be intended to detect obstacleson the road, other vehicles, pedestrians, lane marks, traffic signs,traffic lights, and other road objects. The processing The firstprocessing device can be further adapted to preform image processingtasks which can be intended to estimate a profile of a road along a pathof the vehicle. The path can be an estimated path of the vehicle. By wayof example, the first processing device can be adapted to obtain theinformation mentioned above, or detect the objects listed above, as wellas other objects, exclusively or non-exclusively based on monocularimage processing. As mentioned in this disclosure, while it is possiblethat monocular imaging will be used in embodiments of the presentinvention, it is also possible that stereo imaging would be used andalso that other types of sensors, including, for example, radar, LIDARand kinematic sensors, for example, would be used.

Still further, the first processing device may calculate a disparity ofpixels between the images from the main camera and the narrow camera (orany other pair of stereo setup) and create a 3D reconstruction of theenvironment of vehicle 200. The first processing device may then combinethe 3D reconstruction with 3D map data (e.g., a depth map) or with 3Dinformation calculated based on information from another camera.Optionally, the first processing device can be configured to use thetrained neural network on depth information (for example the 3D mapdata) to estimate a future path ahead of a current location of avehicle, in accordance with examples of the presently disclosed subjectmatter. In this implementation the neural network can be trained ondepth information, such as 3D map data.

The second processing device may receive images from main camera and canbe configured to perform vision processing to detect obstacles on theroad, other vehicles, pedestrians, lane marks, traffic signs, trafficlights, and other road objects. Additionally, the second processingdevice may calculate a camera displacement and, based on thedisplacement, calculate a disparity of pixels between successive imagesand create a 3D reconstruction of the scene (e.g., a structure frommotion). The second processing device may send the structure from motionbased 3D reconstruction to the first processing device to be combinedwith the stereo 3D images or with the depth information obtained bystereo processing.

The third processing device may receive images from the wide FOV cameraand process the images to detect obstacles on the road, vehicles,pedestrians, lane marks, traffic signs, traffic lights, and other roadobjects. The third processing device may execute additional processinginstructions to analyze images to identify objects moving in the image,such as vehicles changing lanes, pedestrians, etc.

In some embodiments, having streams of image-based information capturedand processed independently may provide an opportunity for providingredundancy in the system. Such redundancy may include, for example,using a first image capture device and the images processed from thatdevice to validate and/or supplement information obtained by capturingand processing image information from at least a second image capturedevice.

In some embodiments, system 100 may use two image capture devices (e.g.,image capture devices 122 and 124) in providing suspension controlassistance for vehicle 200 and use a third image capture device (e.g.,image capture device 126) to provide redundancy and validate theanalysis of data received from the other two image capture devices. Forexample, in such a configuration, image capture devices 122 and 124 mayprovide images for stereo analysis by system 100 for vehicle 200, whileimage capture device 126 may provide images for monocular analysis bysystem 100 to provide redundancy and validation of information obtainedbased on images captured from image capture device 122 and/or imagecapture device 124. That is, image capture device 126 (and acorresponding processing device) may be considered to provide aredundant sub-system for providing a check on the analysis derived fromimage capture devices 122 and 124.

One of skill in the art will recognize that the above cameraconfigurations, camera placements, number of cameras, camera locations,etc., are examples only. These components and others described relativeto the overall system may be assembled and used in a variety ofdifferent configurations without departing from the scope of thedisclosed embodiments. Further details regarding usage of a multi-camerasystem to provide driver assist and/or autonomous vehicle functionalityfollow below.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications can be madeto the foregoing disclosed examples. For example, not all components areessential for the operation of system 100. Further, any component can belocated in any appropriate part of system 100 and the components can berearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and control vehicle 200 orsystems thereof or alert a user of the vehicle in response to theanalysis.

As discussed below in further detail and according to examples of thepresently disclosed subject matter, system 100 may provide a variety offeatures related to suspension control autonomous driving,semi-autonomous driving, and/or driver assist technology. For example,system 100 can analyze image data, position data (e.g., GPS locationinformation), map data, speed data, and/or data from sensors included invehicle 200. System 100 may collect the data for analysis from, forexample, image acquisition unit 120, position sensor 130, and othersensors. Further, system 100 can analyze the collected data to determinewhether or not vehicle 200 should take a certain action, and thenautomatically take the determined action without human intervention orit can provide a warning, alert or instruction which can indicate to adriver that a certain action needs to be taken. Automatic actions can becarried out under human supervision and can be subject to humanintervention and/or override. For example, when vehicle 200 navigateswithout human intervention, system 100 may automatically control thesuspension, braking, acceleration, and/or steering of vehicle 200 (e.g.,by sending control signals to one or more of throttling system 220,braking system 230, suspension system 245 and steering system 240).Further, system 100 can analyze the collected data and issue warningsand/or alerts to vehicle occupants based on the analysis of thecollected data.

Reference is now made to FIG. 4, which is a flowchart illustration of amethod of providing a road profile along a predicted path of a vehicle,according to examples of the disclosed embodiments. It would beappreciated that the method illustrated in FIG. 4, and described hereinwith reference thereto can be implemented on the system 100 shown inFIG. 1, and can be part of the system shown in FIG. 2F. However, itshould also be noted, that the method illustrated in FIG. 4, anddescribed herein with reference thereto, can be implemented on any othersuitable hardware and can be implemented as part of any suitable system,in particular as part of any suitable vehicular system that isconfigured to use input related to a road profile.

Returning now to FIG. 4, in block 410, a plurality of images of an areain a vicinity of a user vehicle may be obtained. Optionally, the imagescan be obtained from one or more image capture devices (e.g., cameras),such as image capture device 122, image capture device 124, and imagecapture device 126, mounted on or in the user vehicle.

One or more predicted paths of the user vehicle can also be obtained(block 420). The predicted paths may be provided as input or can beestimated, for example, by software running on computer hardware, suchas on the system 100 shown in FIG. 1. In other examples the predictedpath is estimated, at least in part, on a remote and is communicated tothe vehicle as input possibly with some additional processing takingplace in system 100 onboard the vehicle.

A predicted path of the user vehicle may be determined in various ways.In one example, the predicted path may be determined by reading acurrent heading of the vehicle and predicting that the vehicle willmaintain the same heading or predicting the future path of the vehiclebased on its current heading, and optionally also based on its headingin the past (e.g., the vehicle heading trend over a duration of a fewseconds or in previous drives along the same path). The heading of thevehicle can be obtained, for example, from kinematic sensors onboard thevehicle, such as steering angle sensors, gyroscopes, yaw sensors, etc.In determining the predicted path an instantaneous heading and/or aheading trend over a duration, of say a few seconds, can be obtained andused.

The current heading of the vehicle can be obtained from the steeringsystem of the vehicle, for example. In an electronic steering system,the current heading can be obtained directly from the controller of theelectronic steering system. In other types of steering systems, a sensormay be used to obtain a current state of the steering system or one ofits components, and the current heading can be determined from thesensor's output. In yet further examples, the heading of the vehicle canbe obtained from gyroscopes or gyrostabilizers onboard the vehicle, thatare capable of providing at least a yaw angle and/or a yaw rate of thevehicle. It would be appreciated that other sensors which are capable ofproviding a yaw angle of the vehicle can be used, possibly incombination with other data, to determine a predicted path of thevehicle.

In another example, the path of the vehicle can be predicted from imagesof an environment of the vehicle.

Optionally, images of the road ahead of the vehicle can be processed,and lane markings appearing in the images and/or road edges, such ascurbs, can be detected in the images. Various models can be used inprocessing the images for estimating a predicted path of the vehicle.For example, image based lane detection methods, such as the ones used,for example, by Lane Departure Warning (LDW) systems can be used.Optionally, if lane marks are detected in the images it may be predictedthat the user vehicle will keep on traveling within the lane, at thesame distances from the lane marks. In a further example, the pathprediction method may be configured to predict a small shift back to thecenter of the lane or some other typical position for a particular useror for a particular vehicle or for a particular combination of userand/or vehicle. The prediction algorithm can also be configured tosupport cutting of curves, such that the predicted path is shifted fromthe center of the lane, by a certain extent or at a certain rate, on (atleast some) curves. The maneuver computed by the path predictionalgorithm and which is represented by the predicted path, can bepredefined, can take into account the layout of the curve, itscurvature, the elevation map of the curve, user behavior andpreferences, presence of other vehicles on the curve, the specific laneon which the vehicle is predicted to be when entering the curve (in casethere is more than one lane), characteristics of the vehicle, itsmaintenance condition, user/driver policy and settings, the weather,time of day, etc.

Optionally, images captured by the image acquisition devices onboard theuser vehicle can also be processed to determine lane merges or splits,highway exists, and other path related image features and such imagefeatures can also be taken into account when determine a predicted pathof the vehicle. Based on the lane markings and/or road edges detected inthe images, a predicted path of the vehicle can be determined.Optionally, when predicting the path of the vehicle based on lanemarking and/or road edges which are detected in images of the road aheadof the vehicle, a position of the vehicle can also be determined. Thus,for example, in a multi-lane road, if the vehicle is currently travelingwithin a certain lane, the prediction may assume that the vehicle willstay within the same lane. In other examples, the prediction can alsoestimate which lane the vehicle will be on in the future, and this maynot necessarily be the same lane as the one currently the vehicle iscurrently on, for example, when there is a lane merge ahead, or when thelane that the vehicle is currently on is congested. Examples of methodsof detecting lane marks, road, edges, and lane merges in images, and ofpredicting a path of vehicle based on lane marks detected in images andbased on and other image data are disclosed, for example, in thefollowing commonly assigned patent applications, publications and issuedpatents: U.S. Provisional Patent Application No. 62/270,431, PCTApplication No. PCT/US16/17411, U.S. Pat. Nos. 9,233,688, 9,205,835,U.S. patent application Ser. No. 14/828,112, US Patent ApplicationPublication No. 2014/0160244, each of which is hereby incorporated byreference in its entirety. Mobileye Vision Technologies Ltd. ofJerusalem, Israel markets products that implement these and other objectdetection algorithms, including vehicles, pedestrians, animals, etc. byprocessing images captured by a camera mounted onboard a vehicle.

According to examples of the presently disclosed subject matter, anothermethod which can be used to predict a path of the user vehicle which canis based on machine learning. Optionally, a neural network can betrained, over a large set of images, to predict, from an image of anenvironment of a user vehicle, the path that the user vehicle will take.Optionally, the machine learning learns from the images the actual pathtaken by the vehicle/driver determined by looking ahead in the imagesequence to see where the vehicle actually traveled. A neural networkcan be thus trained to learn the correct path taken by thevehicle/driver, where the cost function for training can be, forexample, the distance between the predicted path and the actual path.The distance can be measured as pixels in the image or lateral distanceon the road. An example of a machine learning based path predictionmethod is disclosed in the commonly assigned U.S. Provisional PatentApplication No. 62/181,784, which is hereby incorporated by reference inits entirety.

The machine learning method can be used to predict the path for usercontrolled vehicles, for autonomous vehicles or for both. The machinelearning based path prediction method can be combined with any of otherpath prediction methods. For example the machine learning based pathprediction can have different neural networks for different types ofusers/drivers. Thus for example, for some users the neural network canbe trained to predict more aggressive lines through curves. In still afurther example, the machine learning based path prediction can havedifferent neural networks for different types of vehicles, such thattrucks and private cars are trained, at least to some extent, withdifferent paths through similar paths.

According to a further example of the presently disclosed subjectmatter, the path prediction algorithm can also take into account apresence of an object on the road. Still further by way of example, incase an obstacle (e.g., a speed bump or a pothole) is detected on theroad, the path prediction algorithm may be configured to estimate thelateral extent of the obstacle. According to one example, an obstacledetection process which can be implemented by the system can includeprocessing the images acquired by the image capture device onboard thevehicle can to detect a dark (or bright) patch associated with theobstacle, and the lateral extent of the path in the image can beestimated. At night protruding obstacles, such as speed bumps forexample, may be detected in the image by a brighter horizontal patch onthe near side of the bump due to illumination of the car headlight on asurface which is more upright, and by a darker patch on the far side ofthe bump dues to the lack of illumination by the host car headlightswhich are lower than the camera. The lateral extent of these light anddark patches gives the lateral extent of the bump and suggest analternative path. A similar method can be devised for other types ofobstacles including recessed obstacles, such as potholes.

The detected obstacle (or a portion of the obstacle) can possibly beprojected to the real world to determine its real-world measurements.Optionally, if the lateral extent is not large, say less than apredefined threshold, the path prediction algorithm can be configured topredict a path for the user vehicle that bypasses the obstacle to theleft or right. Thus, for example, if a speed bump is detected ahead ofthe vehicle, and it is determined that the speed bump does not cover thefull width of the lane it might be reasonable to predict that the driverwill aim for a path that will bypass the speed-bump. If a speed bump (orany other obstacle) is detected in the images, a search can be made inthe image from the high point on the bump (or the low point in case of apothole, or the extremity, in general) laterally to estimate an end tothe bump along the row. Another option if a bump (or obstacle) isdetected, would be to test whether a bump exists along a path close tothe edge of the lane, and in particular close to a road boundary, suchas a sidewalk. If no bump exists at the edge of the lane it might bereasonable to predict a path where one wheel stays on the level road,avoiding the speedbump (or obstacle).

For example, a (relatively) computationally efficient alternative pathsearch can involve: detecting a deviation from smoothness (can bedefined in various way as discussed herein) along a current (predicted)path. If excessive deviation from smoothness is detected at a certaindistance the alternative smooth regions can be searched for at thatdistance. If such a “smoother” region is detected, an alternative path,starting from the current path, passing through the smoother alternativearea, and retuning to the current path can be constructed, and may beused as a replacement or an additional predicted path. Optionally, itcan also be estimated whether a lateral deviation from the path wouldcause less (or more) discomfort than passing over the obstacle.Optionally, the alternative path can then be analyzed for safety andnearby vehicles and obstacles. Lane type information can also be used toensure that the alternative path does not cross a solid white line orget too close to a curb. Optionally, the alternative path analysis orone or more steps of the process can be partitioned between the roadprofile computation system and the steering control system. If thealternative path is determined to be better than the previous predictedpath it can be provided as output to the steering controller.Optionally, since computing a profile can be computationally expensivethe system can be configured to first evaluate whether the alternativepath is acceptable and/or desirable based on less computationallyintensive processes, such as processes which are used to determineeffects, availability, bounds or ability of lateral motion, safety ofcertain maneuvers or a more abstract safety envelope, etc.

In some cases an obstacle in or on the road surface has no telltaleindication in the image texture. For example, badly laid road or frostheaves can produce such smooth bumps. By way of example, in this or inother cases where the system is configured to do so, an efficient searchfor alternative paths can be performed. By way of example, according tosuch a search method, once a bump is detected along the predicted pathan alternative path of the maximal allowable lateral deviation can beexplored. If this alternative path turns out to be smooth a third pathcan be explored in between the first and second paths. If this thirdpath is significantly smoother (the determination of smoothness isdiscussed elsewhere in this disclosure), say than the original, a fourthpath can be explored between the 3rd and first paths. This form ofbisection can be performed until no better improvement in smoothness orlateral deviation is expected, at which point the alternative isselected as the new predicted path. Given the finite width of thevehicle's tire there is a limit to the effect of minor changes in thepath. The extent of potholes can similarly be detected and alternativepaths can be evaluated and used to control the vehicle's path to avoidthe wheel going into the pothole.

Optionally, driver/user habits or driver/user policy can be used,possibly in combination with other path prediction methods, to predictwhich path a user will choose for the vehicle. If, based on pastbehavior of the user often tries to bypass the bump, then this behaviorcan be detected, recorded and then used when predicting a path for theuser vehicle.

Another indicator that can be used to determine a predicted path is yawrate. In particular yaw rate extracted from images, such as the imagesacquired by the image acquisition unit onboard the user vehicle. Ingeneral, yaw rate can be noisy but if it is determined that there is abypass path and the yaw rate indicates a turn towards the bypass then itraises the probability that the bypass will be taken. If the requiredsteering to perform the bypass is not large and the path is clear fromroad edge or other vehicles the likelihood that the bypass will be takenis also greater. Thus, for example, multiple factors can be combinedtogether and can be used to determine a predicted path or to select fromtwo or more possible predicted paths: required turn for new path, safetyof new path, yaw towards new path and driving habits. The yaw rate canalso be used to determine and record the driver policy, mentioned above,although any other suitable method can be used for determining thedriver policy.

Optionally, when a path is predicted in response to a detected obstacleahead of the vehicle, the predicted path can be computed based on,possibly among other factors, traffic conditions. In this regard, itwould be appreciated that the driver will be less likely to swerve toavoid a pothole if there is a vehicle in the adjacent lane. As mentionedabove, surrounding traffic and other safety conditions can be taken intoaccount in combination of as part of path prediction processes that arebased on other methods.

According to another example of the presently disclosed subject matter,the path of the user vehicle can be determined from a trajectory or apath, either a prestored trajectory or path, or one that is calculated,for example, based on a prestored layout or trajectory of the road or ofindividual lanes along a certain road. The path can take into accountone or more of the following: a current location of the vehicle, asegment of road on which the vehicle is located, a lane in which thevehicle is positioned, a starting point of the vehicle, a destination ofthe vehicle, and a suggested route of the vehicle (e.g., out of variouspossible routes from a given source point to a given destination point,the route which the user vehicle is predicted to take).

It would be appreciated, that a known route (out of various possibleroutes from a given source point to a given destination point, the routewhich the user vehicle is predicted to take), may not be of sufficientresolution for use in determining an effective road profile estimate fora user vehicle. One reason is that many obstacles on the road do notextend across the entire route, and at a given point along a route, adriver or a control unit of an autonomous vehicle can select differentpaths, and along such different path the road profile can vary. Thus,for active or adaptive suspension systems a more refined, higherresolution path prediction may be required. One possible path predictionmethod which can be used is the crowdsourced based trajectory estimatemethod as suggested for example, in the commonly assigned PCTApplication No. PCT/US16/17411. This method is based on a sparse map, tosupport low-bandwidth updates. However, it would be appreciated thatother methods, including methods that use HD-Maps and other forms ofpre-stored paths can be used.

In accordance with another example, the path prediction can be performedtaking into account user specific input, or input that is specific tothe type of vehicle or even the specific vehicle for which the path isbeing predicted, the user (e.g., a human driver or humanoperator/passenger in case of an autonomous vehicle) riding the vehiclefor which the path is being predicted. The user/vehicle specific pathprediction input can relate to the driving behavior or policy of theuser or vehicle. For example, some users can be more prone to “cutting”corners, which means they follow a more “aggressive” lines through turnskeeping to the outside edge of a lane when initiating the turn,gradually approaching the inside edge of the lane and the apex of theturn, and the gradually approaching the outside edge of the lane againat the turn exit. In another example, some types of vehicles can beassociated with “sweeping” sharp turns, such as buses and trucksmaneuvering sharp turns, such as can be often found in urbanenvironments. Such inputs relating to driver policy, user (past)behavior, vehicle type, maneuver capabilities and characteristics of thedriver or of the vehicle can be provided as input to a path predictionalgorithm and can be taken into account when computing the estimatedpath of the user vehicle.

Another form of input data that can be used in determining a predictedpath is user input. For example, a plurality of paths can be presentedto the user and the user can select a preferred path. The selectionoperation can be carried out through any available user interface,including for example, a touch (on a tactile screen), a voice command(through a voice recognition/command), by a hand gesture (using a cameraand image processing software), etc. In another example, the eyemovement of the user can be tracked (using a camera) and the pathprediction algorithm can analyze the user's direction of (gaze) focus toestimate a predicted path, possibly with some of the other pathprediction method mentioned herein.

According to examples of the presently disclosed subject matter, morethan one (e.g., two, three, . . . , n) paths can be predicted and aplurality of such predicted paths can be obtained. As will be describedbelow, the system computes a road profile along a predicted path, andoptionally, the system can compute a plurality of road profiles for arespective plurality of predicted paths, where at least one of the roadprofiles is computed based on image data. The system can provide theplurality of road profiles as output, and let the controller, say anactive suspension controller, decide which one of the plurality ofpredicted paths is closest to the most up to date path estimate, and usethe respective road profile. It would be appreciated that predicting aplurality of paths, and computing a plurality of road profiles may bemore computationally economical than computing a full road surfaceprofile, and leaving it to the suspension controller to figure out therelevant portion of the road.

According to examples of the presently disclosed subject matter, aplurality of path prediction methods can be used and evaluated overtime, and the best one (or two, three, . . . , or n methods) can be usedfor predicting the vehicle's path, and along which predicted paths theroad profile can be computed, as described above. Still further by wayof example, one method that can be used to evaluate the quality oraccuracy of a path prediction method can involve comparing the predictedpath resulting from a respective method with the actual path recordedfor the vehicle and measuring a lateral distance between the two pathsat a certain distance or at a certain headway, for example, in seconds,or computing the mean lateral distance or the number of frames where thelateral distance was greater than a certain threshold such as one tirewidth. In another example, such a measure can be computed only forsequences where there are road bumps or other places where thesuspension controller actually needed to take action.

According to examples of the presently disclosed subject matter, one ormore preferred path prediction methods can be selected, and the use ofone or more prediction methods can be suspended or discontinued. Forexample, if it is determined that a certain method seldom matches thepath that is actually used by the user vehicle, or is too far apart(based on some measure) from the path that is actually used by the user,that method can be suspended from further use in the future, or it canbe used less often. Still further by way of example, one or moreprediction methods which have been suspended or discontinued may bere-evaluated from time-to-time. Still further by way of example, it ispossible that one path prediction method will be selected and only suchsingle path prediction method shall be used, e.g., for a particularvehicle, user, area, type of curves, etc., or combinations thereof.

The predicted path can take on any form and can be provided in anysuitable reference frame. For example, the predicted path can beprovided as a spline in an arbitrary coordinate frame, such as disclosedin the commonly assigned PCT Application No. PCT/US16/17411. In anotherexample, a predicted path can be denoted by a plot of 2D or 3D points ina global reference frame. In case a plot of 3D points is used to denotethe predicted path, the z coordinates (elevation) may not have asufficient density (e.g., the sample points may be too far apart) or thez coordinates may not have a sufficient resolution, or the quality ofthe data may not be sufficient (e.g., too noisy, or not accurateenough), so that the estimated road profile along the predicted path canbe used to provide better density and/or resolution and/or quality. Thehigher density and/or resolution and/or quality may be a requirement of,say, an active or adaptive suspension system. The predicted path canalso be provided with an indication of a width of an obstacle along thepredicted path, or on (either) side of it. In another example, the imagebased road profile can simply provide redundancy and backup in case thepredicted path also includes elevation data.

Reference is now made to FIGS. 5A-5E show an image of a road ahead of auser vehicle and two predicted paths overlaid thereon, according toexamples of the presently disclosed subject matter. Lines 502A-502E and504A-504E denote a right and left wheel tracks, respectively, of apredicated path that is based on steering wheel angle. Lines 506A-506Edenote a (single) predicted path that is based on machine learning, forexample, based on the method disclosed in U.S. Provisional PatentApplication No. 62/181,784. It would be appreciated that the predictedpath 506A-506E can be converted to provide a predicated path along thevehicle's wheel tracks, in a similar manner to the steering angle basedpredicted path denoted by lines 502A-502E and 504A-504E, based on aknown relative position of the camera onboard the user vehicle and thevehicle's wheels.

As can be seen in FIGS. 5A-5E, at least in this scenario, the steeringwheel angle based path prediction, denoted by lines 502A-502E and504A-504E, is correct on straight lines (see for example FIG. 5D) andinside the curve (see for example FIG. 5B), but at the beginning and endof curves the steering wheel angle based predicted path is incorrect(see for example FIGS. 5A and 5B). The machine learning based pathprediction, denoted by lines 506A-506E, correctly predicts the curvesahead throughout the scene.

As will be described below, and with reference to one possible scenariofor the example shown in FIGS. 5A-5E, the system according to examplesof the presently disclosed subject matter can compute a road profile forone or both of the predicted paths using image data. For example, thesystem can be configured to process the images shown in FIGS. 5A-5E andcompute a road profile along one or both of the predicted paths. In oneexample, the processor can determine, instantaneously or over a periodof time, that the machine learning (in this example) based pathprediction is more accurate, or the processor may determine that themachine learning (in this example) based path prediction is moreconsistently accurate enough, and the imaged based road profile can becarried out only along the path that was predicted using machinelearning. As mentioned above, there can be various possibleimplementations for a process of selecting which path predictionmethod(s) to use as part of examples of the presently disclosed subjectmatter, and various variations based on additional inputs, environmentalconditions, user behavior and preferences, characteristics of the road(such as type of curve, etc.), safety and other conditions of thevehicle's surroundings, etc.

Resuming now the description of FIG. 4, at block 430 the plurality ofimages of the area in the vicinity of the user's vehicle can beprocessed, a profile of a road along one or more of the predicted pathsof the user vehicle can be estimated based on image data. According toexamples of the presently disclosed subject matter, as part ofestimating a profile of a road along a predicted path, possibly alongeach predicted path from a plurality of paths along which a road profileis estimated, a wheel track or wheel tracks along the path can bedefined or estimated. The wheel track, can include one or more estimatedwheel tracks of the user vehicle along the respective predicted path.Still further by way of example, the profile of the road along each oneof the one or more predicted paths of the user vehicle can be limited toapproximately the width of the track of each wheel or wheel-pair (orgroup of wheels for trucks and other such multi-wheeled vehicles) of thevehicle along the respective predicted path. The wheel track can bebased on generic information or it can be based on information that isspecific to the user vehicle, such as location of the wheels relative tothe camera or any other sensor or reference point onboard the vehiclethat is used to determine the position of the vehicle along thepredicted path. Similarly, the width of the predicted path or ofportions of the predicted path can also be associated with a measured,generic other estimated width of the user vehicle's wheels or tires.Thus for example, for a four-wheel car, the predicted path can includetwo wheel tracks at the location of each pair of the car's wheels, andthe width of the path can be equal (or similar) to the width of thecar's tires.

One example of a method that can be used for processing the plurality ofimages of the area in the vicinity of the user's vehicle to compute aprofile of a road along a predicted path of the user vehicle isdescribed in the commonly assigned U.S. Pat. No. 9,118,816, which ishereby incorporated by referenced in its entirety. U.S. Pat. No.9,118,816 discloses a driver assistance system that is operable, whilethe host vehicle is moving, to detect a deviation in vertical contour ofa road. A first image frame and a second image frame are captured in thefield of view of the camera. Image motion is processed betweenrespective images of the road derived from the first image frame and thesecond image frame. The vertical contour of the road is estimated usinga road surface model, and the deviation in the vertical contour can becomputed from the road surface model. Other road profile formats canalso be used.

According to examples of the presently disclosed subject matter, one ormore predicted paths can be projected onto the image(s) reference frame,and the road profile estimation can be carried out along the predictedpath(s). If the predicted path is already given in the image referenceframe, the projection of the predicted path on to the image referenceframe can be avoided. The computation of the road profile can thus belimited to a predicted path or paths, reducing the computational loadand other resource consumption, latency, etc.

In another example, the processor can be configured to implement a modelbased classifier to detect certain obstacles. For example, the processorcan be preconfigured to process image data and detect image data whichcorresponds to Berliner Cushions on the road. Optionally, the processorcan also use prestored specifications of certain obstacles, such thatwhen such obstacles are detected, their dimensions and possibly furtherinformation such as their rigidity, etc., can be quickly and easily(without much computation load) determined. Optionally, the availabilityof bypass maneuvers or possibility of avoiding such known obstacles canalso be predefined or prestored and can be relatively easily andefficiently be access by the processor for computing a road profile orfor determining an (alternative) predicted path.

It would be appreciated that other image based methods can be used todetermine a road profile along a predicted road, including monocularbased image processing methods, other image flow analysis methods,and/or methods which are based on stereoscopic image processing, andparticularly processing of images taken from two or more cameras, butalso time of flight and active triangulation methods that use aprojector to project a pattern onto a scene (including the roadsurface), and capture and process an image of the a reflected portion ofthe projected pattern.

In a two camera system a known relative translation and orientation ofthe two cameras together with an estimated or predicted road plane canbe used to determine the homography between the two images. The residualmotion after alignment using the homography can assist in determiningthe road profile.

It would be further appreciated that other methods can be used incombination with image based road profile estimation method(s), toestimate the profile of the road along a predicted path, including forexample, methods that are based on Lidar systems, which can be used toscan a surface of the road to determine the road's profile, radar basedsystem's etc.

Reference is now made to FIG. 6, which is an example of an output roadprofile along a predicted path, in accordance with examples of thepresently disclosed subject matter. On the left side, an image acquiredby a camera attached to an inner surface of a front windshield of a uservehicle is shown, and on the image a predicted path with residual motionis shown. On the top right, there are shown current road profiles alongeach of a left and a right wheel track of the vehicle along a predictedpath from 5 m to 20 m ahead of the vehicle. The predicted path issampled every 5 cm from 5 m to 10 m, and every 10 cm from 10 m to 20 m.On the bottom right, there is shown a 1D sample for each of a left andright wheel tracks of a predicted path showing the accumulated profilefrom 7 m ahead of the vehicle to the vehicle itself (or some point ofreference on the vehicle). In this example, the predicated path isstraight ahead, and just the one predicted path is shown. However, itwould be noted, and shown below, that other scenarios occur and arehandled the method and system according to the present disclosure. Itwould be appreciated that the parameters presented in FIG. 6, areprovided by way of example only, and that other parameters or otherdensities, resolution, etc. can be used in examples of the presentlydisclosed subject matter.

It would be appreciated that due to latency of the communication andcomputation into, inside and from the camera unit, a predicted path canbe determined based on old information, and in the case of maneuversmight be incorrect. If the predicted path does match the actual paththen it is possible that the suspension system will respond to roadbumps, pot holes, and other obstacles on the road that the wheels willnot actually encounter. This is of particular importance with activesuspension which lift and lower the wheels. The suspension controllercan be configured to compare the path used for computing the profilewith the most current path estimation available. In addition, accordingto the examples of the presently disclosed subject matter, the predictedpath and the road profile that is computed based on the predicted pathcan be continuously updated, providing the suspension controller withupdates. If, however, there is not enough overlap between the predictedpath and the actual path, or if it is otherwise determined that the roadprofile that is computed by the system according to examples of thepresently disclosed subject matter, does not match the actual currentroad profile which the suspension controller is required to react to,then the suspension controller can ignore the road profile informationand work as a purely reactive system. It would be appreciated thatexamples of the presently disclosed subject matter suggest possible pathprediction methods that can achieve better accuracy of predictionrelative to the (simplistic) methods that are based solely or mostly onthe actual path traveled by the vehicle, e.g., using steering angle, oryaw sensors.

The format of the road profile output can be provided in any formsuitable format, for example, it can be in a format that is compatiblewith a suspension control unit onboard the vehicle. In one example, theroad profile output can be provided as an elevation map. In anotherexample, the road profile output can be provided as two streams ofelevation maps, one per each wheel track of the vehicle. The roadprofile data can be provided for a given distance ahead of the vehicle.The distance to which the road profile data can be fixed or can vary. Incase the distance to which the road profile relates is variable, theroad profile output can also include a spatial or temporal reference,which would indicate to a suspension control system to determine whichis the appropriate time to use or act upon the road profile datareceived from the system. In yet another example, the system can beconfigured to re-evaluate the accuracy and/or the relevance of the roadprofile output which it previously provided and can issue updates orindications that the old data was not accurate or correct. In suchcases, the suspension controller may be configured to rely on the morerecent data or can choose which data to rely on. In another example, theroad profile data can relate to a certain range ahead of the vehicle,and can be continuously updated with new data which overlaps at least tosome extent some of the old (recent) data.

According to examples of the presently disclosed subject matter, theroad profile output format can include an indicator as to the width ofan obstacle detected on or near a predicted path with which the roadprofile is associated. For example the road profile output can includean indication that a pothole along the predicted path, with which theprofile is associated, extends 0.75 m to the left and 0.5 m to the rightof the predicted path. Having this indication can allow a controllerwhich receives the road profile output to determine that when (or solong as) the actual path is within those margins the road profileinformation should be used to control the suspension and/or othersystems or components of the vehicle, which may be affected by thisparticular road profile. Likewise, if the actual path is outside themargins, the specification of the obstacle which was provided with theroad profile output can enable the controller to ignore it, and, forexample, switch the vehicle's suspension to reactive mode.

Still further by way of example, the road profile along a predicted pathcan be provided in a stabilized coordinate frame. The profile can thenbe output as a profile on a frame by frame basis as with the previousversion or the profile can be sampled to produce the road height at aparticular point in front of the wheel (possibly for each wheel or wheelpair) at a high sample rate. This latter format may require a lower rateof CAN messages (e.g. 10 samples per frame rather than 200 samples inother implementation) and may require less processing on the receivingside (e.g., the suspension system controller).

According to a further aspect of the presently disclosed subject matter,there is provided a control system for an advanced driver assistancesystem (ADAS) that includes an active steering function. Optionally theADAS system is an autonomous vehicle (or a control system for anautonomous vehicle. In another example, the ADAS system includes a lanekeep assist function (LKA). According to examples of the presentlydisclosed subject matter, the control system for an autonomous vehiclein accordance with examples of the presently disclosed subject matter,can include an image capture device, a data interface and at least oneprocessing device, configured to receive images captured by the imagecapture device through the data interface. The processing device can beconfigured to obtain a path of the vehicle, and the processing devicecan be configured to compute a road profile along the predicted path.The path of the vehicle can be determine, at least in part, based onimage data. It would be appreciated that a system which includes anactive steering function, such as an AV system or an LKA system orfunction, can have a path determining/fetching function as an integralpath thereof.

Optionally, the processor can be configured to estimate a smoothness ofa first path, and the processor can search for a second path that issmoother than the first path. The smoothness can be evaluated in variousways. For example, the average elevation modulation can be taken intoaccount. In another example shifts in elevation that are larger than acertain extent can be more severely regard (e.g., can be associated withhigh negative scores), certain uneven patterns can have a higher impacton the smoothness evaluation/score, etc. Optionally, the processor canbe configured to compute a smoothness score for a path which is providedby the autonomous vehicle system (say, by the steering control system),where the score can be computed based on one or more factors, such asthe ones mentioned above, and possibly others, and when the score is notsatisfactory (say, below a predefined threshold), the processor cantrigger or carry out itself an alternative path prediction process, inan attempt to find an alternative smoother path. As mentioned above,there are various methods that can be used to predict a path for a uservehicle and any such suitable method can be used. If the second (orthird, or fourth, etc.) path is selected, the steering control system,and possibly the suspension control system and any other relevant systemof the autonomous vehicle can receive as output the selected path andpossibly also the road profile along the selected path.

It would be appreciated that the path prediction and the road profilecomputation, in particular along a certain path, can be carried outaccording to the examples described above. Optionally, the alternativepath prediction process can be controlled so that the alternative pathis distant from the rejected path(s) by at least a certain margin. Anexample of such a margin can be a wheel width or the user vehicle. Inanother example, the margin can be based on an analysis of the lateralextent of an obstacle on the road, as was described above. In anotherexample, the road and/or lane boundary can constrain the alternativepath search. In yet another example, the search for an alternative pathcan simply involve a search for the closest path that is smooth enough(using any of the criteria mentioned here or any other suitablecriterion), and leave it to the steering controller to determine if sucha path deviation may be considered. Similar considerations may be usedto steer around a pothole or to ensure that the pot hole passesunderneath the center of the vehicle.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, or other opticaldrive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A system, comprising: at least one processingdevice configured to: receive a plurality of images of an area in avicinity of a user vehicle captured by an image capture device;determine a first predicted path of the user vehicle based on analysisof the plurality of images; determine a second predicted path of theuser vehicle based on a heading of the user vehicle; and compute a firstvertical contour of a road along the first predicted path and a secondvertical contour of the road along the second predicted path; and outputa control signal configured to modify an operation of a component of theuser vehicle based on one of the first vertical contour and the secondvertical contour.
 2. The system according to claim 1, wherein theprocessing device is further configured to compute a yaw rate of theuser vehicle by estimating an ego-motion of the vehicle from theplurality of images.
 3. The system according to claim 1, wherein theprocessing device is further configured to identify lane markings in theplurality of images in the vicinity of the user vehicle, and theprocessing device is configured to determine a location of the uservehicle relative to the identified lane markings.
 4. The systemaccording to claim 1, wherein the second predicted path is predictedbased on a kinematic sensor onboard the user vehicle.
 5. The systemaccording to claim 4, wherein the kinematic sensor is a sensor adaptedto measure a yaw rate of the vehicle or a steering angle of the vehicle.6. The system according to claim 1, wherein the processing device isfurther configured to detect an obstacle along at least one predictedpath of the user vehicle, and wherein the processing device isconfigured to compute a third predicted path which avoids the obstacle.7. The system according to claim 6, wherein the obstacle is detectedbased on a difference in smoothness along at least one of the firstvertical contour and second vertical contour.
 8. The system according toclaim 1, wherein the processing device is further configured to computeat least one of the first vertical contour and the second verticalcontour along estimated wheel tracks along at least one predicted pathof the user vehicle.
 9. The system according to claim 8, wherein atleast one predicted path has a width substantially equal to the width ofthe wheel tracks of the user vehicle.
 10. The system according to claim1, wherein the processing device is further configured to: determinewhich of the first predicted path or second predicted path is moresimilar to an actual path of the user vehicle; and select one of thefirst vertical contour and the second vertical contour based on thedetermination; wherein the modification of the operation of thecomponent is performed based on the selection.
 11. The system accordingto claim 1, wherein the first vertical contour and the second verticalcontour are associated with at least one temporal reference specifying atime to modify the operation of the component.
 12. A method, comprising:receiving a plurality of images of an area in a vicinity of a uservehicle; obtaining a first predicted path for the user vehicle based onanalysis of the plurality of images; obtaining a second predicted pathfor the user vehicle based on a heading of the user vehicle; computing afirst vertical contour of a road along the first predicted path and asecond vertical contour of a road along the second predicted path; andoutput a control signal configured to modify an operation of a componentof the user vehicle based on one of the first vertical contour or thesecond vertical contour.
 13. The method according to claim 12, furthercomprising computing a yaw rate of the user vehicle by estimating anego-motion of the vehicle from the plurality of images.
 14. The methodaccording to claim 12, further comprising identifying lane markings inthe plurality of images in the vicinity of the user vehicle; anddetermining a location of the user vehicle relative to the identifiedlane markings.
 15. The method according to claim 12, wherein the secondpredicted path is predicted based on information received from akinematic sensor onboard the user vehicle.
 16. The method according toclaim 12, further comprising detecting an obstacle along at least onepredicted path of the user vehicle; and computing a third predicted pathwhich avoids the obstacle.
 17. The method according to claim 16, whereinthe obstacle is detected based on a difference in smoothness along atleast one of the first vertical contour and second vertical contour. 18.The method according to claim 12, further comprising computing at leastone of the first vertical contour and the second vertical contour alongestimated wheel tracks along at least one predicted path of the uservehicle.
 19. The method according to claim 18, wherein at least onepredicted path has a width substantially equal to the width of the wheeltracks of the user vehicle.
 20. The method according to claim 12,further comprising: determining which of the first predicted path orsecond predicted path is more similar to an actual path of the uservehicle; and selecting one of the first vertical contour and the secondvertical contour based on the determination; wherein the modification ofthe operation of the component is performed based on the selection. 21.The method according to claim 12, wherein the first vertical contour andthe second vertical contour are associated with at least one temporalreference specifying a time to modify the operation of the component.